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Preface 


Over the past decade, the Texas Legislature passed several bills that required colleges to begin developing 


and scaling corequisite models of developmental education (among other reforms).' For example, House 

Bill 2223 required that 75 percent of developmental education enrollment be in corequisite remediation by 
fall 2020 (with the exception of a few exempted student groups), so colleges around the state have priori- 
tized corequisite remediation to ensure that these requirements are met. Corequisite remediation shifts the 
way that developmental education is provided so that students directly enter a college-level course and 
receive additional academic support during that same semester to address the challenges they face with 
basic math, reading, or writing concepts. The state and institutions have rolled out substantial professional 
development resources to support practitioners in response to developmental education reform. Data-driven 
improvement efforts were identified as a top priority for professional development, and this toolkit was 
developed as a resource for these professional development efforts. 


This toolkit was undertaken by RAND Education and Labor, a division of the RAND Corporation that conducts 
research on early childhood through postsecondary education programs, workforce development, and pro- 
grams and policies affecting workers, entrepreneurship, and financial literacy and decisionmaking. The Texas 
Higher Education Coordinating Board partnered with RAND and American Institutes for Research to support 
the state’s developmental reform efforts through several studies designed to evaluate those efforts, funded 
by the U.S. Department of Education’s Institute of Education Sciences. The work on these studies informs the 
content of this toolkit. The first study (supported by grants R305H170085 and R305H150094) included a rig- 
orous randomized control trial of corequisite remediation in five community colleges and a statewide study of 
how corequisite remediation was implemented across Texas.” This work provided us with an understanding 
of how corequisite remediation was commonly rolled out in colleges and insight into the decisions around 
program delivery and the potential issues that colleges may encounter. The second study (supported by 
grant R305H150069) focused on close technical assistance work with six Texas community colleges to help 
them engage in improvement processes with developmental education reform initiatives. This work provided 
us with an understanding of how best to carry out improvement efforts in college settings. In addition to 
“on-the-ground” experiences with colleges in Texas, these studies provided us with opportunities to review 
the literature on improvement approaches and accelerated models of developmental education. 


The research reported here was supported, in whole or in part, by the Institute of Education Sciences, 

U.S. Department of Education, through grants R305H150069 and R305H150094 to the RAND Corporation 
and R305H170085 to American Institutes for Research. The opinions expressed are those of the authors and 
do not represent the views of the Institute of Education Sciences or the U.S. Department of Education. 


More information about RAND can be found at www.rand.org. Questions about this toolkit should be direct- 
ed to Idaugher@rand.org, and questions about RAND Education and Labor should be directed to 
educationandlabor@rand.org. 


Is This Toolkit for? 


This toolkit was developed for improvement teams made up of administrators, department and faculty chairs, and other 
college staff who are helping to oversee important initiatives and programs—such as corequisite remediation—and who 
are looking to build an ongoing process for addressing challenges they face and improving the way that corequisite mod- 
els are provided through data use and reflection. 


Is the Purpose of the Toolkit? 


This toolkit aims to act as a guide to improvement strategies for teams of practitioners at colleges, providing them with 
the knowledge and tools they need to carry out rapid cycles of data-driven improvement. 


Is in this Toolkit? 


In this toolkit, we provide an overview of some key strategies for making quick improvements to educational programs 
and some tools and tips for carrying out these efforts on the ground. We focus on a common reform being adopted by 
colleges: corequisite remediation. For more information on what corequisite remediation is and why it is an area in which 


improvement strategies may be useful, see Box A. 


Sections of the toolkit include the following: 


Approaches to improvement. A brief overview of two approaches to improvement—quality improvement and 
rapid-cycle evaluation—that colleges can use to adopt, deliver, and scale corequisite models. 

The value of improvement approaches to colleges. A discussion of how improvement strategies call for a different 
and valuable approach to rolling out programs and initiatives, such as corequisite remediation. 


Identifying the focus of improvement efforts. A description of some initial efforts required to prepare for improve- 
ment cycles focused on corequisite remediation. 


Carrying out quality improvement through a Plan-Do-Study-Act (PDSA) cycle. A guide to a process for testing in- 
cremental improvements to the delivery of corequisite remediation. In addition to describing the PDSA process, this 
section includes tips and tools for (1) identifying questions and making predictions, (2) assigning roles and responsi- 
bilities, and (3) developing and sticking to a time line. 


Carrying out rapid-cycle evaluation. A guide to a process that can be used to evaluate the effectiveness of particular 
corequisite model components, practices, or processes before scaling. The section includes tips and tools for (1) deter- 
mining what to measure, (2) identifying the appropriate data sources, and (3) using comparison groups. 


Ensuring success in improvement efforts. Some concluding thoughts on some of the most important keys to suc- 
cess for improvement efforts in colleges. 
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Approaches to Improvement 


In education, practitioners are increasingly looking for strategies to bring data and 
evidence into decisionmaking and to roll out programs in intentional ways that allow 


for ongoing improvement. The hope is that these data-driven approaches will lead to 

educational programs and practices that better support the success of students. This 

toolkit provides processes and strategies drawn from two fields: quality improvement 
and rapid-cycle evaluation. We provide a brief introduction to each of these fields and 
their approaches to improvement below. 


Quality Improvement 


The field of quality improvement grew out of work from a series of researchers who 
argued that the scientific process should be conducted with a cyclical rather than 
straight-line approach.? These cyclical approaches, such as the PDSA cycle, were 
adopted widely by practitioners to make incremental improvements to manufacturing 
processes and health care services, and they have more recently become more pop- 
ular in education settings.* The idea was that organizations could encourage regular, 
ongoing improvement by identifying potential areas for advancement, testing changes 
ona small scale, examining data to determine whether they were successful, and then 
gradually scaling and adjusting these changes through many 

rapid cycles of improvement. 


Quality improvement also 
Experts built on the idea of testing changes through rapid PDSA P 
cycles by developing an overarching model that guides quality places q strong emphasis 


improvement through three questions: on a democratic approach, 

+ What are we trying to accomplish? whereby practitioners 

¢ How will we know that a change is an improvement? are driving the work and 
are regularly engaging 


understanding of the problem, quality improvement calls for to reflect throughout the 


the testing of changes through a series of PDSA cycles that lead process. 

to incremental improvements and ongoing data collection and 

reflection until the problem is sufficiently addressed. Quali- 

ty improvement also places a strong emphasis on a democratic approach, whereby 
practitioners are driving the work and are regularly engaging to reflect throughout the 


« What changes can we make that will result in improvement?® 


After a college addresses these questions and develops a clear 


process. 
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Practitioners 
needed more 
rigorous ways 
to assess 
certain program 
components as 
they were rolled 
out. 


Quality improvement is a useful approach for colleges to pursue when they 
want to develop a better understanding of why they are facing particular 
issues in delivering corequisite remediation, and when colleges want to 
test out small, incremental changes to address the issues that they face 
and scale them over time. This toolkit provides some important strategies and 
processes for quality improvement (see the sections on Identifying the Focus 
of Improvement Efforts and on Carrying Out Quality Improvement Through a 
Plan-Do-Study-Act Cycle). 


Rapid-Cycle Evaluation 


Although such fields as education and health care have long used program evalu- 

ation to assess impact and implementation, high-quality program evaluation has 

traditionally been long and resource-intensive and therefore limited in its use for 

quick decisionmaking about how to develop and scale programs and practices. 

However, practitioners needed more rigorous ways to assess certain program 

components as they were rolled out. In response, experts in evaluation developed 
the practice of rapid-cycle evaluation and began to use this approach to 
evaluate the effectiveness of health care technology, education technology, 
and program delivery.® Rapid-cycle evaluation makes comparisons between 
individuals who receive some process, practice, or program feature with 
individuals in a comparison group in order to evaluate the effectiveness of 
that process, practice, or program feature. 


Rapid-cycle evaluation is the best approach when colleges want to 
understand whether a particular corequisite remediation process, prac- 
tice, or feature is effective in order to help with decisionmaking about 
adopting, scaling, modifying or eliminating the practice, process, or 
component. This toolkit describes some important strategies and tools for 
rapid-cycle evaluation (see the sections on Identifying the Focus of Improve- 
ment Efforts and on Carrying Out Rapid-Cycle Evaluation). 
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What Is Corequisite Remediation 
AND WHY INCORPORATE IMPROVEMENT STRATEGIES? 


Many students enter college with an assessment of being “not college ready” in math, reading, or writing. Traditional- 
ly, colleges have required these students to enroll in a series of developmental education courses that must be com- 
pleted prior to entering college-level courses. Data from 2010 suggest that more than two-thirds of community college 
students enrolled in at least one developmental education course.’ 


Corequisite remediation shifts the way that developmental education is provided so that students directly enter a col- 
lege-level course and receive additional academic support during that same semester to address the challenges they 
face with basic math, reading, or writing concepts (Figure A.1). Corequisite models vary in the way that they deliver 
academic support and may include paired courses, more instructional time to supplement the college course time, 


workshops, required use of tutoring, or office 
Figure A.1. Corequisite Remediation Versus Traditional Developmental Education 


6-}@ 


hours.® Corequisite remediation aims to improve 
college persistence and completion by increas- 
ing student momentum in earning college credits 
and better aligning the basic skills support that 
students receive with college-level coursework. 


TRADITIONAL 


A national survey in 2016 found that 35 percent 


of two-year colleges were offering corequi- 


site remediation for reading and writing, and 
EMIC SUp 


= 


There are many areas of uncertainty for colleges and states about how to design and scale effective corequisite models, 


16 percent were offering corequisite remediation 
for math.’ These numbers will increase substan- 
tially, as state policymakers and large college 

systems in such states as Tennessee, Texas, and 


COREQUISITE 
REMEDIATION 


California have since approved sweeping policy 
reforms that require colleges to scale corequisite 
remediation to most students who enter college with a “not college ready” assessment.”° 


and, as is true of many new reforms, colleges have encountered some challenges as corequisite remediation has been 
rolled out to students (e.g., staffing considerations, challenges with scheduling and advising)." There are several rigor- 
ous studies that examine the impact of corequisite remediation and provide promising evidence that this approach can 
improve student success.” However, there is limited information on how colleges have dealt with common problems or 
which aspects of the corequisite models were most important to their success. 


States, college systems, and associations increasingly are offering professional development to build knowledge 
around best practices for corequisite remediation, but professional development sessions often are based on anec- 
dotal evidence and may or may not be relevant to the settings and corequisite models of other colleges. Data-driven 
improvement efforts can support and complement these professional development efforts by helping colleges develop 
more-systematic evidence on their promising practices and address real issues they face in their own contexts. 
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Differences in Approaches 


As the descriptions above highlight, rapid-cycle evaluation and quality improve- 
ment are similar in that they can be used to address important problems faced by 
practitioners through the use of short cycles of inquiry that require collaboration, 
experimentation, data use, and reflection. However, these approaches also differ 
in some important ways, including the following: 


¢ Starting point. In rapid-cycle evaluation, the college starts with a particular 
practice, process, or program component and wants to evaluate its effective- 
ness (i.e., its impact on outcomes). In quality improvement, the college starts 
with a set of improved outcomes it wants to achieve and uses the process to 
identify and test possible approaches toward those outcomes. 


Scope of improvement. Although rapid-cycle evaluation is often used to test 
the effectiveness of an entire practice or process or a key program compo- 
nent (as well as smaller changes), quality improvement focuses exclusively on 
smaller changes to a practice, process, or program. Rapid-cycle evaluation 

is sometimes a one-shot deal and other times involves multiple cycles of 
evaluation, while quality improvement assumes that many short cycles may 
be required to test out several incremental changes. 


Emphasis of measurement efforts. Because rapid-cycle evaluation is 
focused on measuring impacts, it requires rolling out programs in ways that 
allow for strong comparison groups, collecting high-quality data on outcomes, 
and engaging in rigorous analysis of the data. In quality improvement, there is 
a greater focus on collecting simple, real-time measures that can be com- 
pared across many different improvement cycles and engaging practitioners 
to reflect and respond to results. 


Individuals involved in improvement efforts. It will be important for col- 
leges to include evaluation experts on the improvement team when conduct- 
ing rapid-cycle evaluation because of its greater emphasis on rigorous impact 
analysis. With quality improvement, it is critical to ensure practitioner-driven 
change and reflection, so teams may pull in a broader group of practitioners 
who are willing to engage in unpacking the challenges the college is fac- 

ing, supporting real-time data collection, reflecting on results, and scaling 
improvements. 


These two approaches to improvement are complementary, and colleges can 
draw from both approaches to address problems they face on the ground related 
to corequisite remediation. For example, a college may conduct a rapid-cycle 
evaluation of a new software product it is considering using in a corequisite mod- 
el. After determining the product is effective, the college may then use quality 
improvement strategies to support faculty as they integrate the software into the 
classroom and make ongoing alterations. Colleges should think carefully about 
which approach to improvement will be most useful for addressing their particular 
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problem and focus on just one approach at a time when carrying out a cycle of improve- 
ment efforts. 


The Value of Improvement 
Approaches to Colleges 


There are many reasons why improvement approaches that rely on rapid cycles can 
be valuable in helping colleges develop and scale corequisite models, as well as other 
college programs and initiatives. We highlight five of the benefits that improvement 
strategies offer colleges, as follows: 


¢ Are applied across many contexts, programs, and initiatives. The improvement 
strategies we discuss in this toolkit are adaptable and can be used for all types of 
programs and initiatives (e.g., developmental education reforms, guided pathways 
initiatives) and all types of colleges (e.g., two-year or four-year, small or big). 


Help colleges identify and address issues that they face. Colleges often face 
uncertainty about what to expect when they roll out new programs and initiatives 
and encounter challenges that prevent them from delivering and scaling programs 
successfully. Improvement strategies provide tools for prioritizing challenges and 
tackling them in an orderly fashion by rolling out programs in deliberate ways to 
inform improvement.” 


Ensure colleges use data to drive program improvement. Data use and insti- 
tutional research in colleges are often driven by reporting requirements for state 
and federal agencies, accreditation bodies, and funders. Improvement strategies 
provide the opportunity to collect and analyze data in ways that address internal 
priorities determined by the college and help colleges develop valuable real-time 
measures that can be acted on immediately." 


Do not necessarily require complex research methods. Colleges vary in their 
research capacity and access to external research support, and they do not always 
have the resources to conduct extensive data collection and use rigorous research 
methods to assess programs and initiatives. Although data use is a central feature 
of improvement efforts and must be prioritized, some approaches to improvement 
do not require substantial research expertise or resources. 


Facilitate long-term planning and improvement. It can be challenging to roll out 
the perfect program, and colleges may not do so optimally the first time. Improve- 
ment strategies can help ensure that colleges approach the rollout and refinement 
of programs in a more deliberate way through experimentation, data use, and 
reflection to ensure smart decisionmaking and improvement over the long term." 


Although improvement approaches can provide many benefits to colleges, they also 
require resources. A focus on data-driven decisionmaking and improvement requires a 
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A problem of 
practice is an issue 
that a college is 
Struggling with or 
an area in which it 
would like to make 
improvements. 


shift in culture that may need sustained leadership and commitment over a long 
period. In addition, each time a college engages an improvement cycle, this pro- 
cess requires the time of key administrators, faculty, and other staff; resources to 
support data collection and analysis; and the dedication of leadership to support 
change. For colleges that face constraints in resources, it may not be appropriate 
to build cycles of improvement into every program and initiative, and colleges may 
instead need to focus on high-priority initiatives. 


Identifying the Focus of 
Improvement Efforts 


Prior to engaging in any improvement effort—whether through a quality improve- 
ment or rapid-cycle evaluation approach—it is important for a college to clearly 


determine the focus of those efforts, including the specific problem related to 
corequisite remediation that the college aims to address and the specific practic- 
es, processes, or corequisite model components that will be tested or evaluated 
through rapid cycles of inquiry. In this section, we describe the process of identify- 
ing a problem of practice and objectives for improvement cycles. 


Identifying the Problem of Practice 


The first step in planning for an improvement cycle is to identify a problem 
of practice. A problem of practice is an issue that a college is struggling with 
or an area in which it would like to make improvements. We provide some 
examples of common problems of practice that colleges have faced related 
to corequisite remediation in Table 1. 


Colleges may have a variety of issues they need to tackle related to coreq- 
uisite remediation, so choosing a single problem of practice to focus on for 
improvement may be challenging. Our experiences and the improvement 
literature suggest that colleges should consider the following four criteria 
when prioritizing among the various problems they face: 


¢ There is a consensus among key stakeholders that the problem of 

practice is a high priority. In selecting a problem of practice, colleges 
should choose problems for which there is general agreement among admin- 
istrators, faculty, and other relevant staff that the problem is an important one 
to address. This consensus ensures that individuals will be willing to engage 
in the various activities required to carry out improvement strategies and 
willing to make decisions and changes based on the evidence that emerges. 
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¢ The problem of practice is under the purview of the college. 
The problem of practice must be something that individuals on the The problem of 
improvement team have the power to change. Problems of practice 2 
like “issues with high schools not sufficiently preparing students” practice must be 


or “challenges students are facing with life events that are causing something that 

them to drop out” are not ideal because they are problems that are Peete 

largely outside of the control of the college. individuals on the 
¢ Addressing the problem of practice will drive student success. im p rovement team 

Student success in terms of increased learning, degree comple- 

tion, and preparation for employment is central to the mission of a have the power to 

college. If addressing the problem of practice will not substantial- h 

ly contribute to the college’s mission, the problem should not be C ange. 


prioritized. For example, uncertainty about whether to assign a letter 

grade or a pass-or-fail determination for the academic support por- 

tion of the corequisite model may be an issue that faculty are struggling with, but if 
this decision has few implications for the longer-term success of students, it should 
not be prioritized. 


The problem of practice cannot be adequately addressed through existing 
evidence. As colleges continue to adopt and scale corequisite remediation, the 
evidence base on effective models and promising practices is likely to grow. Col- 
leges should first look to what is known internally and what evidence is available 
about what other colleges have done to support successful corequisite models. 
There may still be value in gathering local evidence to complement evidence from 
the literature; however, all else being equal, problems of practice for which there is 
little or no existing evidence might be prioritized." 
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Table 1. Examples of Problems of Practice and Improvement Cycle Objectives for Corequisite Remediation 


Problem of Practice 


Sample Objectives for 
Quality Improvement 


Sample Objectives for 
Rapid-Cycle Evaluation 


Uncertainty about the best way to deter- 
mine which students should be placed 
into a corequisite model 


Test a new checklist for advising 
students into the corequisite 
model 


Evaluate the success of a stu- 
dent group (e.g., lower-scoring 
students) in the corequisite 
model 


Challenges assigning instructors and 
scheduling linked sections of the college 
course and academic support 


Test a new faculty and advisor 
communication protocol 


Evaluate the impact of a process 
to streamline scheduling 


Challenges enrolling students into linked 
sections of the college course and aca- 
demic support 


Test an information technology 
fix that links courses for the pur- 
poses of course registration 


Evaluate the impact of an enroll- 
ment auditing process 


Uncertainty about how to build faculty 
capacity to teach in corequisite models 


Test a new training guide 


Evaluate the effectiveness of 
a new training requirement for 
instructors 


Concerns about collaboration and align- 
ment when corequisite models are taught 
by instructor pairs 


Test a requirement to have 
aligned syllabi 


Evaluate the impact of having 
a single instructor versus two 
instructors 


Challenges streamlining content to 
provide sufficient academic support with 
less instructional time 


Test a streamlined lesson plan 
for a particular concept 


Evaluate the impact of a corequi- 
site model with an extra hour of 
academic support 


Uncertainty about whether certain fea- 
tures of a corequisite model (e.g., class 
size, same instructor) are essential to 
maintain for student success 


Test a small modification to how 
the corequisite model is deliv- 
ered 


Evaluate the effectiveness of 
the corequisite model with and 
without a key feature 


Uncertainty about how to ensure that a 
corequisite model is effective for stu- 
dents with particular characteristics (e.g., 
lower placement scores, ESL students) 


Test an instructional strategy 
geared toward supporting the 
student subgroup 


Evaluate the effectiveness of 
the corequisite model for the 
student subgroup 


Uncertainty about the best way to serve 
students who have been unsuccessful in 
the corequisite model 


Test new guidance provided 
to instructors on strategies for 
individualized support 


Evaluate the effectiveness of a 
new individualized corequisite 
remediation approach 


NOTE: A worksheet for prioritizing problems of practice is provided in a supplement at the end of this toolkit; ESL = English as a second 


language. 
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Choosing an Objective for the Improvement Cycle 


Once a problem of practice has been chosen and the college knows which approach to 
improvement it will pursue, the next step is to identify a single objective for a specific 
improvement cycle. This objective should be something that the college hopes will 
lead to clear progress in addressing the problem of practice and 

improving corequisite remediation. Both rapid-cycle evaluation 


and quality improvement call for improvements to be tested or This objective should be 


evaluated in a short period (90 days or less for quality improve- A 
ment and up to nine months for rapid-cycle evaluation), so a something that the college 


college needs to choose a change to corequisite remediation that hopes will lead to clear 
can be tested, or some aspect of corequisite models that can be progress in addressing 


the problem of practice 


evaluation efforts are closely related (Table 1), as both approach- and Improving co requ Isite 
es aim to test or evaluate something and gather evidence to remediation. 
determine whether that improvement to an aspect of corequisite 


evaluated, within this brief time frame. 


The objectives for quality improvement efforts and rapid-cycle 


remediation should be scaled, modified, or abandoned. How- 

ever, as noted earlier, the scope of what is being tested or evaluated may differ across 
the two improvement approaches, and quality improvement calls for a more thorough 
process of identifying the changes that will be tested. We describe how objectives are 
determined for each of the two approaches in the following sections. 


Choosing an Objective for Rapid-Cycle Evaluation 

In rapid-cycle evaluation, colleges typically come with something predetermined 

that they would like to evaluate, so the process of choosing an objective is relatively 
straightforward. However, it is important to ensure a clear vision of how conducting 
that evaluation will lead to improvement on the problem of practice. For example, if a 
college must rely both on instructor pairs and on single instructors to lead instruction 
across the course and academic support because the pool of instructors is limited, it 
may not be very useful to improvement efforts to conduct an evaluation comparing the 
effectiveness of two-instructor models and one-instructor models. Instead, it might 
be useful to evaluate methods of improving instruction regardless of how models are 
staffed. On the other hand, if a college has more staffing flexibility and does want to 
choose a single approach to staffing corequisite models, an evaluation comparing the 
two might be more appropriate. 


Choosing Objectives for Quality Improvement 
With quality improvement, a college does not start with something it would like to test 
or evaluate. Instead, it must engage in a more thorough and structured process for 
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The field of quality 
improvement calls 
on practitioners to 
develop a deeper 


understanding of what 
is driving the problem 


in order to identify 
changes that might 


determining what will be tested through PDSA cycles. The college improvement 
team must first define a model for improvement that will help them figure out 
what they want to achieve through the improvement process and develop a clear 
road map toward that desired improvement. Defining a model of improvement 
requires a college to identify the following items: 


1. Aim: What is the college trying to accomplish? 
2. Measures: How will the college know that a change is an improvement? 


3. Changes: What changes can the college make that will result in improvement?” 


Determining a problem of practice is a first step to identifying the aim 

of improvement. The aim of improvement should also clearly state what 
progress on the problem is expected and when. For example, if a col- 
lege chooses “enrolling students in linked learning community sections 
of a corequisite model” as its problem of practice, it might develop the 
following aim statement: “We aim for 95 percent of the students enrolled 
in corequisite remediation to be coenrolled in the correct, linked sections 
of the college course and academic support by fall 2021.” 


To address the second question, colleges will need to think carefully 
about whether they need to look at processes, outcomes, or both to 
assess improvement and to think about specific measures that can be 
used throughout the improvement process to track differences across 


lead to Improvements. the changes that are being tested. We provide more detail on selecting 


measures later in the toolkit. 


The third question calls for practitioners to develop a deeper understanding 

of what is driving the problem in order to identify changes that might lead to 
improvements. By better understanding these “drivers,” a college will be able to 
identify improvements that will help solve the problem. Each possible change can 
then be set as an objective and tested through a PDSA cycle. 


There are many strategies and tools that practitioners can use to identify drivers 
and develop changes that can be tested through PDSA cycles. One common 
approach is a structured team process called root cause analysis, which uses a 
cause-and-effect diagram—or “fishbone” diagram—to map out the drivers of the 
problem. An example of a fishbone diagram is presented in Figure 1. The process 
for completing a fishbone diagram includes the following steps: 


¢ Write the problem at the mouth of the “fish.” 


« Define the different categories of things that might cause that problem (e.g., 
people, policies). 


« Ask “Why does this happen?” to identify different causes of the problem, and 
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record these causes on the “bones” of the fish 

Identify subcauses underlying the initial set of causes recorded in the diagram, 
and keep breaking the issue down until participants cannot think of any additional 
causes. 


Select (or rank) the causes for which participants see the most potential improve- 
ment, and list out specific improvements that can be tested through improvement cy- 
cles. Each improvement that will be tested is the objective for that improvement cycle. 


Root cause analysis typically incorporates the voices of the full range of individuals who 
have roles or perspectives on the problem of practice, which may include administrators, 
faculty, other school staff, and external stakeholders. Common strategies for gathering 
the input of many people are to use sticky notes to identify causes and to ask participants 
to vote for the top causes for which they believe improvements should be made. 


Figure 1. Example of a Fishbone Diagram Describing the Problem of Student Enrollment in 
Linked Sections of the Corequisite Model 
College 
administrators, 
faculty, 
Students support staff 


Large — . 
caseloads Limited advisor 


capacity 


Jobs, families 
limit availability 


PROBLEM: 
High turnover 
Students do not rates Challenges 
follow advisor No single admin getting 


recommendations overseeing process 


students into 
linked sections 
of the college 


No requirement for students Beadiict course and 
to meet with advisors before limitations Software does 
course registration not link courses developmental 
education 
Linked course numbers Lack of support 


staff time to 
input course 
numbers 


often sent after 
registration starts 


Rules, Facilities 
policies, and 
procedures supplies 


NOTE: A worksheet for completing root cause analysis is provided in a supplement at the end of this toolkit. 
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A PDSA cycle can be 
a valuable tool whe 
colleges are interes 


in improving corequisite 


remediation by test 


Prior to embarking on a rapid cycle of improvement, the college will have identi- 
fied an objective for the improvement cycle, either (1) an incremental improvement 
the college wants to test and refine to address the problem (quality improvement) 
or (2) a particular corequisite model component, process, or practice that needs to 
be evaluated to address the problem (rapid-cycle evaluation). In the next section, 
we discuss how to carry out a PDSA cycle to address quality improvement objec- 
tives, followed by a description of the rapid-cycle evaluation process. 


Carrying Out Quality 
Improvement Through a 
Plan-Do-Study-Act Cycle 


A PDSA cycle can be a valuable tool when colleges are interested in improving 


corequisite remediation by testing out incremental changes in the way that core- 

quisite remediation is delivered. In this section, we first describe the four stages 
of a PDSA cycle (Figure 2) and provide an illustration of what a 
PDSA cycle might look like if carried out on a common problem of 
practice related to corequisite remediation (Box B). Then we pro- 

n vide some tools and tips on (1) identifying questions and making 
predictions, (2) assigning roles and responsibilities, and (3) staying 


ted on track. 


At the outset of any improvement efforts, the college should have 
Ing out identified an improvement team to lead the planning process and 


incre mental cha nges in oversee the activities in the PDSA cycle. This improvement team 
should also have led the process of developing a comprehensive 

the way that corequisite understanding of the problem of practice and determining the ob- 

programs are provided. jective for the PDSA cycle. This team will have the ultimate respon- 


sibility for ensuring that evidence from the cycle translates into 

action. This improvement team is likely to oversee multiple PDSA 

cycles until a particular problem of practice has been addressed 
and the college moves on to a new problem that calls for a different team make- 
up. We provide additional tips in a later section on what to consider when building 
a strong improvement team (Tips and Tools: Getting People Involved). 
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Figure 2. Key Activities in a Plan-Do-Study-Act Cycle 
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Key Activities in the Plan Stage 


The Plan stage is about figuring out what the PDSA cycle will focus on and setting 
up a plan that will help a college successfully carry out the cycle. We describe 
five types of activities that commonly take place in the Plan stage: (1) identifying 
questions to be addressed in the cycle and making predictions, (2) mapping out 
measures and data that will be used to assess improvement, (3) defining the 
activities that will take place throughout the cycle, (4) assigning roles and respon- 
sibilities, and (5) laying out a detailed time line. 


It is important to define questions that will drive the data collection and analy- 
sis based on the specific objective for a PDSA cycle and make predictions that 
will be tested in the cycle. For example, if the objective is to pilot a new checklist 
for advisors on enrolling students into linked sections of the college course and 
academic support, one question that a college might aim to answer could be 
“Were there fewer underfilled and overfilled sections after implementing the new 
scheduling process?” The improvement team should make a specific prediction, 
such as “There will be a 10-percent decrease in the number of sections that were 
underfilled or overfilled.” Later in this toolkit, we provide additional guidance on 
identifying questions and making predictions (Tips and Tools: Identifying Ques- 
tions, Predictions, and Measures). 


As part of the process of defining questions and making predictions, colleges will 
also need to map out a set of measures and data that will be used to assess 
progress toward improvement. These measures should be easy-to-access, 
real-time measures that can be tracked as improvements are rolled out. Using the 
same example, to assess whether the new checklist for corequisite remediation 
advisors was an improvement, the college may want to look at data on how often 
the checklist was completed, the length of advising sections, whether sections 
were underfilled or overfilled, and student satisfaction with the advising process. 
Quality improvement experts often call for a common set of measures that are 
determined outside of the PDSA cycle to allow for comparison of progress across 
many different PDSA cycles in an improvement effort, though there may also be 
additional measures that improvement teams choose to collect that are particular- 
ly informative for certain types of improvements. 


Next, the college must define the activities for the other stages of the PDSA 
cycle, and the college must be able to reasonably accomplish such activities 
within a short period (ideally 90 days or less per PDSA cycle). These activities will 
be guided by the specific objective(s) and questions that the college has chosen 
for the PDSA cycle. For example, if a college chooses an objective of testing out 
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a new corequisite remediation checklist, the college will need to lay out a clear plan 
for what needs to be done to roll out this checklist (“improvement activities”) and also 
a plan for collecting the data on advising sessions, student enrollment, and student 
satisfaction (“evaluation activities”). 


Once a college has identified the key activities that will take place in the cycle, it is criti- 
cal for the improvement team members to assign roles and responsibilities to them- 
selves for overseeing these activities. They must also consider others outside of the 
group who will need to contribute in various ways to ensure the improvement activities 
are carried out successfully. For example, if a college is testing out its new corequisite 
remediation advising checklist, this group may involve administrators, faculty, advisors, 
the registrar, and students. 


In the final step of the Plan stage, the improvement team should lay out a detailed time 
line for the PDSA cycle to ensure that the various activities stay on track and can be 
completed in a short time. For example, if a college really wants to be scaling improve- 
ments to the corequisite remediation advising process in one year, the improvement 
team will need to lay out a plan for multiple (and possibly simultaneous) PDSA cycles to 
address the problem in the year leading up to that deadline. The process of identifying 
the problem and objectives for improvement can also take several months and needs 

to be built into the time line. We provide some additional tips on the time line in a later 
section of this toolkit (Tips and Tools: Developing a Time Line and Staying on Track). 


Key Activities in the Do Stage 


The Do stage is about delivering the program or initiative and collecting data to under- 
stand how things are going. We describe two types of activities that commonly take 
place in the Do stage: (1) delivering the program or initiative with an improvement that is 
being tested and (2) collecting data to inform improvement. 


First, a college should deliver the specific improvement that is being tested to 
address the problem of practice. In initial PDSA cycles, the college may want to roll 

out a new corequisite remediation advising checklist to a handful of advisors. The 
checklist can be tested at a larger scale in later PDSA cycles. College staff involved in 
delivering the program should have clear guidance and ongoing support to ensure that 
they understand the expectations for how they should be delivering the program. For 
example, guidance and support might include (but is not limited to) documents describ- 
ing improvement activities, a workshop or training that introduces those changes, and 
ongoing check-in calls to answer questions and offer help as needed. 
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Study @ 


Finally, the Do stage requires colleges to collect data to inform improvement as 
the program or initiative is rolled out. Although colleges can continue to assess 
existing student success data, such as course passing rates and persistence, to 
inform improvement, colleges will need to collect data that track progress within 
shorter periods (e.g., within semesters) to allow for quicker assessments of prog- 
ress in rapid PDSA cycles. For example, depending on the corequisite remediation 
problem of practice, colleges might want to collect data on attendance, home- 
work completion, length of advisor visits with students, weekly audits, or course 
registration. Systematic collection of these types of data is not common in many 
colleges, so these efforts are likely to require substantial leadership support and 
strong buy-in from the college staff involved in collecting the data. We provide 
some tips on data collection in the rapid-cycle evaluation section of the toolkit 
that may be relevant to quality improvement efforts (Tips and Tools: Determining 
Which Data to Collect). 


Key Activities in the Study Stage 


The Study stage is the point at which colleges take a look at the findings and 
reflect on them. We describe three types of activities that commonly take place 
in the Study stage: (1) analyzing the data, (2) reflecting on findings with a broad 
group of stakeholders, and (3) documenting what was learned. 


Analyzing the data should be somewhat straightforward, as quality improvement 
calls for descriptive analysis using measures that are easy to collect and can be 
analyzed within weeks of delivering the change. The assessment plan should be 
designed around the specific questions identified in the Plan stage, and analy- 
sis should be conducted to test the specific predictions made for each of these 
questions. In many cases, PDSA cycles might focus on analyzing data only from 
the group testing out the new improvement, so these predictions help establish 
benchmarks for measuring progress (i.e., a standard for knowing whether the 
improvement being tested was successful). Another way to set a benchmark for 
measuring progress is to use a comparison group. Although comparison groups 
are required for rapid-cycle evaluation, they are not as common for quality 
improvement efforts. We provide additional detail on comparison groups later in 
this toolkit in the section on rapid-cycle improvement (Tips and Tools: Using a 
Comparison Group). 


Once the improvement team has had an opportunity to analyze the findings 

on the key questions for the PDSA cycle, these findings should be shared with 
other individuals who might have important perspectives on the findings and the 
individuals who will drive the improvement efforts based on the findings. For ex- 
ample, the improvement team might want to share its findings from its small test 
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of the advising checklist with a broader group of advisors and may want to pull in the 
registrar and a few faculty to also weigh in. Reflecting on findings with a broad group 
of stakeholders will help ensure that the improvement team did not overlook important 
perspectives on the findings that are evident to a broader set of individuals involved in 
program delivery. In addition, the opportunity to engage with the findings and provide 
input may help build buy-in among the individuals who are essential to driving improve- 
ment. For example, it can take a lot of time and effort from college staff to scale a new 
scheduling process, so it is essential to share findings from a pilot of a new scheduling 
process with advisors, faculty, and the registrar’s office to gather their perspectives and 
demonstrate to them that it is worth their effort to pursue these improvements. 


Finally, the college should make sure to clearly document the findings from the PDSA 
cycle so that future cycles can build from what has been learned. For a simple approach 
to documenting findings, Table 2 in the next section could be adapted to include a 
space for findings. The worksheets provided as a supplement to this toolkit also pro- 
vide a place for documenting efforts. The college could document findings in a more 
comprehensive way, such as in a memo or presentation, in order to share those findings 
with others. 


Key Activities in the Act Stage 


Once the college has identified the problem of practice, gathered some evidence to in- 
form improvement, and reflected on those findings, the final stage of the PDSA cycle is 
to take action to improve the program or policy. The Act stage commonly includes three 
types of activities: (1) determining the changes to be made, (2) sharing findings and 
improvement plans broadly, and (3) identifying questions that require additional study. 


Improvement teams must first determine the changes to be made in response to the 
findings. These actions might include abandoning the change, modifying the change, 
or scaling it as is. For example, if the evidence from piloting the new advising checklist 
for corequisite remediation students shows promise, the college may decide to move 
forward with scaling the checklist to a larger set of advisors for continued testing and 
improvement. Decisions must also be made about how quickly to scale and whether 
additional modifications may be needed prior to scaling. A clear step-by-step plan for 
improvement will help ensure that all relevant stakeholders know what needs to be 
done, who is responsible for doing what, and the time line for making these improve- 
ments. For example, scaling the new advising checklist for corequisite remediation 
may require a handbook that outlines how to use the checklist and training for advisors 
involved in the process. 


In order to mobilize individuals to carry out these improvements and to anticipate 
challenges that might arise, the college should share findings and improvement plans 


a 
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Colleges 
typically 
engage in 
many PDSA 
cycles until 
a problem 
of practice 
has been 
sufficiently 
addressed. 


with a broader group of individuals involved in carrying out these changes. Lat- 
er in this toolkit, we provide some concluding thoughts on conditions that can be 

important to ensuring broad buy-in across stakeholders and driving action toward 
improvement (Conclusion: Ensuring Success in Improvement Efforts). 


In many cases, a PDSA cycle may raise as many questions as it provides answers, 
and colleges should identify questions that require further study. These ques- 
tions can then be addressed in future PDSA cycles. Colleges typically engage in 
many PDSA cycles until a problem of practice has been sufficiently addressed. 
Once the college decides that it has made sufficient improvement on one problem 
of practice, the improvement team can move on to another problem of practice 
related to corequisite remediation. 
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A Fictional Example of a Plan-Do-Study- 
Act Cycle for Corequisite Remediation 


The setting for improvement: A fictional college decides it would like to improve its college algebra corequisite 
model, which pairs a three-hour college algebra course with three hours of weekly academic group support. Last year, 
the pass rate for the college course was just 52 percent, and the college would like 
to make improvements. In this corequisite model, one instructor teaches the college 
algebra course and another instructor teaches the academic support sessions.° 
Instructors are required to use the same textbook and learning objectives, but 
otherwise each instructor has some autonomy to determine the coursework that will 
be covered, the instructional strategies, and the calendar for the two portions of the 
corequisite model. 


A fictional college 
decides it would 


Identifying the problem of practice: In faculty meetings, the instructors teaching 


like to improve its the course have expressed unhappiness with the level of collaboration and coor- 
dination in some instructor pairs. Some instructor pairs work very closely, aligning 
col lege a Igebra their calendars and coursework to ensure tight overlap in content, while other pairs 


have invested relatively little time aligning the two components. Advisors also 
report hearing complaints from students in some sections of the corequisite that 
the course and academic support do not seem to be connected. In addition, some 
students have stopped showing up to the academic support sessions and have told 
instructors that they have decided it is not important to attend as long as they pass 
the college course. After pulling together instructors, advisors, and administrators 
to discuss the situation, there is a consensus that limited instructional alignment 
among some instructor pairs teaching in the corequisite model is an important issue, 
and faculty believe that they can make some substantial improvements in this area. 
The improvement team chooses “challenges with instructional alignment across 
some instructor pairs” as its problem of practice. 


corequisite model. 


Determining objectives for improvement: Because the college does not have a 
predetermined solution to the problem that it can evaluate, it chooses to focus on quality improvement, conduct PDSA 
cycles to test changes, and gradually scale effective approaches over time. The team completes a fishbone diagram to 
understand the sources of the problem and finds that, despite the guidance from the college to closely align instruction 
across the two components of the corequisite model, some instructor pairs have not been engaging in shared planning 
time and have inconsistent syllabi and calendars. After discussing with instructors and getting their input on possible 
changes that could be made to address these issues, the improvement team decides that the objective of the initial 
PDSA cycle will be to test a possible improvement: requiring instructors to spend at least four hours of time together in 
the first few weeks prior to the semester in order to align syllabi and calendars. 


* It is important to note that although this example focuses on issues with a corequisite model that happens to be taught by two instructors, there is no evidence 
suggesting that two-instructor models are any more or less likely to face implementation issues. Furthermore, there is no evidence suggesting differences in student 
outcomes between one- and two-instructor models. A similar process would be valuable for addressing the improvement needs of one-instructor models. 
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Defining the questions, making predictions, and selecting measures: The college also identifies some questions it 
would like to answer through the improvement process and makes specific predictions of what will happen. The college 
outlines four questions it would like to answer: 


¢ Did instructor pairs meet for four or more hours? 
« Did students attend the academic support sessions more often? 
« Did students perceive that the academic support sessions supported their success in the college-level course? 


« Were students more likely to pass the college course? 


If the additional faculty collaboration time has been a success, the improvement team predicts that instructors will have 
met for at least four hours over the semester, that attendance issues will decline by 10 percent relative to prior semes- 
ters, that 75 percent of students will report that the academic support sessions were useful in supporting success in 
their college course, and that pass rates for the college course will have increased by 5 percent. However, because 
course passing will take too long to measure for the rapid 90-day cycles required, the college decides to focus on the 
first three questions. 


Defining activities, assigning roles and responsibilities: The college then maps out what will happen in the PDSA cy- 

cle and who will be responsible for overseeing various improvement and evidence-building activities. For example, the 

college must determine how many instructors will test out the four hours of shared planning time and how to provide 
the guidance needed to support rollout. The college must 


DEANS, | DATA plan for any data-collection activities, efforts to process 
EBCuELY ue EXPERT and reflect on findings, and efforts to act on the evidence. 
pes OTS uc onal “ a i Once the college maps out what will happen throughout 
CO IG CES OTN GEL s : the PDSA cycle, roles and responsibilities can be assigned. 
Analyze student data x The improvement team includes faculty representatives and 
Reflect on findings X X X department chairs, who can help develop the plan for the 
Scale policy if effective x x coordination time and oversee real-time data collection; 


administrators, who have the authority to require and scale 
improvements and can access resources to support the improvements; and staff, who can advise the team on strate- 
gies for collecting and analyzing data. The team members assign roles and responsibilities to themselves for oversee- 
ing each activity and identify others outside of the core improvement team whom they must rely on to carry out various 
activities to support improvement. 


Setting a time line: The team also determines a time line for the PDSA cycle. Planning starts one month prior to the 
semester, and instructors will begin to roll out the change two weeks before the semester starts. Findings will be 
available one month into the 


Sample Time Line of Implementation 
semester, and a deadline is 
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Delivering the program in a way that informs improvement: To test the new instructor collaboration time, the 
college first decides to try out the approach with a small group of four instructor pairs, while other instructor 

pairs continue with “business as usual.” After promising evidence from the fall, the college decides to scale the 
approach to 15 instructor pairs across two campuses in the next round of improvement. Note: /f the college was 
interested in evaluating the effectiveness of the strategy and focused on rapid-cycle evaluation rather than quality 
improvement, it might have wanted to roll out the change to a larger number of instructors, use a lottery approach 
to assign instructors to one group or the other, and focus the analysis on comparisons of these two groups rather 
than using predictions. 


Collecting data: To address the questions posed during the Plan stage and assess whether the improvement has 
been successful, the college collects data over the course of the semester. The college focused on three real-time 
measures: instructor time spent meeting, student attendance, and student perceptions of whether the academic 
support sessions helped to support success in the college course. The improvement team collects weekly in- 
structor surveys to determine how often instructors are meeting under the new requirements and administers a 
short student survey three weeks into the semester. Instructors 
Instructor Survey commit to using the college’s early alert system diligently to track 
e Time spent aligning attendance. 


Student Survey examines the findings from the data. Results indicate that among 


Early Alert System Data 
i e Attendance issues reported Assessing and sharing findings: The improvement team then 
CE * Perceived supportiveness of the four instructor pairs testing the improvement, three did follow 
academic support sessions 

through and meet for the four hours. Students exceeded the 

predicted benchmark for perceptions that the academic support 
sessions increased success in the college course, with 80 percent of all students reporting that this was the case. 
There was a 10-percent decrease in reported attendance issues relative to the prior fall semester. The improve- 
ment team shares these findings with a broader group of instructors and administrators in order to get input, 
ensure that all sources of evidence and perspectives are being considered, and increase the likelihood of buy-in. 


Taking action to make improvements: After sharing the evidence with others in the college, stakeholders within 
the college are in agreement that the alignment strategy should be scaled to 15 instructor pairs in another PDSA 
cycle, with the idea that this strategy may eventually become a requirement for all corequisite remediation instruc- 
tors. The college identifies a faculty task force to oversee the changes and finds the resources to train instructors 
and cover the additional planning time to make sure the changes are sustainable. However, discussions with the 
initial group of instructors who tested out the shared collaboration time suggests that it was really hard to find 
four hours to meet and that three hours may have been sufficient. To address this barrier, the college decides to 
pursue a second PDSA cycle in the spring with a reduction of the requirement to three hours and some additional 
guidance on finding time to meet. Once the college feels comfortable that the improvements have met its expec- 
tations for success, the college can shift focus to other problems of practice. 
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Tips and Tools: Identifying Questions, Predictions, 


and Measures 


This section provides some tips for identifying questions, measures, and predic- 


tions for the purposes of improvement. Rather than casting a wide net to collect 


a great deal of data on a problem of practice and then cutting the data many 


different ways to see what comes out, it is always good practice to lay out specific 


It is always good 
practice to lay out 
specific questions 

that the improvement 
team wants to address 
through data collection 
and analysis in a cycle. 


questions that the improvement team wants to address through data 
collection and analysis in a cycle (e.g., Table 2, which draws from the 
example in Box B). Identifying improvement questions ahead of time 
will also help keep colleges on track with short cycles and will minimize 
the burden of data collection in terms of time and resources. These 
questions should be focused on gathering the critical information that 
must be known to determine whether the change in question is leading 
to improvement. Improvement questions can focus on many different 
aspects of the change that is being tested, including how specific prac- 
tices or processes are being rolled out, the experiences and perspec- 
tives of students and college staff involved in the rollout of the change 
to corequisite remediation, and outcomes for students or staff. 


Table 2. Identifying Questions, Measures, and Predictions 


QUESTION PREDICTION DATA SOURCE MEASURE 
How many hours will Instructors will meet for at Weekly Hours of time collabo- 
instructor pairs subject least 4 hours. instructor rating by phone, email 
to the new alignment survey 
strategy meet for in order 
to align syllabi? 
Did students attend the Reported attendance issues Early alert Number of alerts sub- 


developmental education 
course more often? 


will decline by 10 percent 
from the prior semester. 


system data 


mitted for attendance 
issues 


Will students be more 
likely to perceive that 
the academic support is 
valuable? 


NOTE: A worksheet for documenting questions, measures, and predictions is provided in a supplement at the end of this 


toolkit. 


At least 75 percent of Student Likert question: “The 
students perceive that the survey ad- academic support was 
academic support support- ministered in important to support- 
ed success in the college fourth week ing my success in the 
course. of classes college course.” 
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After defining the questions that will guide data collection and analysis, the college 


must then determine which data and measures will be used to answer the questions. 


Given the emphasis on short cycles and the involvement of a broad range of stake- 
holders, it is wise for colleges to focus on easy-to-collect data and measures that are 


meaningful to many individuals in the college. Quality improvement efforts often aim to 


establish a common set of measures that will be used across multiple PDSA cycles to 
allow for ongoing tracking of progress toward improvement and to allow for compari- 


sons across different types of improvements." We provide additional guidance on data 


sources to consider (see Tips and Tools: Determining Which Data to Collect). 


Improvement teams should also make specific predictions for what they think will hap- 


pen. These predictions will help set clear benchmarks for the college to judge whether 


the change being tested is leading to improvements as hoped. 


Tips and Tools: Getting People Involved 


Our earlier discussion of mapping out roles and responsibilities during the Plan stage 


of a PDSA cycle emphasized two sets of individuals involved in improvement: (1) an 
improvement team with the ultimate accountability for overseeing the process and 


ensuring that evidence-building efforts translate into action and (2) a broader set of 


individuals who might need to be involved in various PDSA activities. Support from 


leadership and a strong emphasis on improvement as a priority for the 
college are also essential to getting individuals across the college to 
engage in improvement efforts. In this section, we provide some tips on 
developing an optimal improvement team, followed by a description of 
how colleges might assign roles and responsibilities across all individu- 
als involved in a PDSA cycle. 


In terms of selecting an improvement team, there is no magic formula 
for the number and types of individuals who should be included, and 
the size and makeup of an improvement team may vary according to 
the context of the college and the problem of practice being addressed. 
However, the literature and our experiences with Texas colleges sug- 
gest that an improvement team should be chosen with the following in 
mind: 


The size and makeup of 
an improvement team 
may vary according 

to the context of 

the college and the 
problem of practice 
being addressed. 


« Team members feel comfortable engaging in open and honest dialogue with all 
other members of the group and are valued equally as participants in the process. 


« Team members are willing to challenge their beliefs and open themselves up to 


new perspectives on issues. 
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Team members have the knowledge and experience to support a deep under- 
standing of the problem of practice. 


Team members have the time and space to meet frequently to oversee the 
improvement process and can devote sufficient time to carrying out their 
roles and responsibilities. 


Team members value data and evidence as an essential tool for developing 
and improving programs and initiatives, and at least some of the team mem- 
bers have some basic understanding of research methods. 


Team members have sufficient authority and resources to drive change in 
support of the problem of practice. 


Team members are committed to student success as the primary focus of 
improvement efforts.” 


We recommend starting with an improvement team of at least three and no more 

than 12 core team members. To ensure that the improvement team has sufficient 

authority to drive change in a college, it will likely need to include administrators. 
Faculty play a critical role in delivering corequisite remediation and 
have important perspectives on most problems of practice related 


The im provement team to corequisite remediation; therefore, most improvement teams are 


should provide room 


likely to require faculty representation. If administrator and faculty 
team members do not have expertise in data collection and research, 
colleges can benefit from bringing on one or more team members who 


for dialogue about 
whether these roles 
and responsibilities 
can be effectively 
carried out with the 
time and resources 
available. 


can provide guidance in this area, such as staff from an institutional 
research office, data specialists supporting college leadership, faculty 
or administrators with a background in program evaluation, and exter- 
nal consultants. And in cases where corequisite remediation problems 
are found outside of the classroom—such as scheduling, advising, 

or tutoring—team members should also be drawn from the relevant 
departments. 


After the improvement team has been chosen, the team may wish to 


spend at least one meeting setting up expectations and group norms 

for the improvement efforts. Such a meeting might include discussion 
of what it means to engage in open, honest dialogue with one another, standards 
or norms for what counts as “evidence” about whether an improvement strategy 
is working, any reporting goals, and the value of varied expertise among those on 
the team. The team may wish to revisit these expectations periodically throughout 
the PDSA cycle. 


The next step is to assign roles and responsibilities for the various activities that 
are planned for a PDSA cycle (e.g., Table 3). Colleges can benefit from assigning 
a single person from the improvement team to assume responsibility for oversee- 
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ing each activity and reporting back to the team on progress, though that accountable 
team member may receive support from others on the team. After assigning team leads 
who hold ultimate accountability for seeing the activities through, improvement teams 
should identify all of the individuals who must be involved with each activity (including 
those outside of the core improvement team), and clearly lay out the specific respon- 
sibilities for each individual or group of individuals. These expectations for roles and 
responsibilities should be clearly communicated to these individuals, and the improve- 
ment team should provide room for dialogue about whether these roles and responsi- 
bilities can be effectively carried out with the time and resources available. Once there 
is common agreement among all of those involved in improvement regarding roles and 
responsibilities, periodic reminders about responsibilities from those overseeing each 
activity and opportunities for feedback on concerns can help provide support. 


Table 3. A Tool for Assigning Roles and Responsibilities 


Students, 
Activity Team Lead | Administrators Faculty Data Specialist Other Staff 
Develop plan Dean Who: Dean, VP of | Who: Faculty lead | Who: N/A Who: N/A 
for aligning Smith Instruction and team 
instruction 
What: Select strat- | What: Design 
: What: What: 
egy strategy options 
Collect data Who: Who: Who: Who: 


Responsibilities: | Responsibilities: 


Responsibilities: 


Responsibilities: 


Analyze data Who: Who: Who: 


Responsibilities: 


Responsibilities: 


Responsibilities: 


Who: 


Responsibilities: 


Scale improve- 
ment strategy 


Who: 


Responsibilities: 


Who: 


Responsibilities: 


Who: 


Responsibilities: 


NOTE: A worksheet for assigning roles and responsibilities is provided in a supplement at the 


end of this toolkit. 


Who: 


Responsibilities: 
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Tips and Tools: Developing a Time Line and Staying 


on Track 


This section provides some tips for developing a time line for improvement. 


PDSA cycles are typically short to encourage rapid feedback and growth (ideally 


90 days or less), while the full process of investigating the problem of practice 


and engaging several PDSA cycles might take a year or more. The reason for 
keeping PDSA cycles short is to encourage organizations to drive changes at a 


quicker-than-usual pace and build on improvement through multiple, iterative cy- 


cles. However, with the many different responsibilities that administrators, faculty, 
and other college staff have outside of improvement efforts, it can be difficult to 


get anything done in these short windows of time, especially 
with efforts that involve the perspectives of many stakehold- 


The reason for keeping ers and the collection and analysis of new data. Colleges must 


PDSA cycles short is to 


have a clear and intentional plan to ensure that the activities 
set out for a PDSA cycle stay on track. As noted previously, 


encourage organizations to choosing “bite-sized” objectives that can reasonably be exam- 
drive changes ata quicker- ined and addressed in a short time is important. 
than-usual pace and build Once a college has set the objective for a cycle and the 


on improvement through 


improvement team has detailed all of the activities that need 
to take place in each of the four stages of the PDSA cycle, 


multiple, iterative cycles. developing a time line is relatively straightforward. Colleges 


should set some key deadlines and check-in points for each of 
the activities in the cycle, and they should develop processes 


for informing everyone involved in improvement efforts about these deadlines 


and hold team leads accountable for sticking to them. There are several options 
for how a college can start to map out the time line for improvement efforts, as 


follows: 


Start from the end. A college may have deadlines for when it would like 

to see improvements and when decisions need to be made to scale those 
improvements. In this case, the college can start by identifying when it would 
like to commence PDSA cycles and work backward to map out the time line. 


Start from the beginning. A college may have constraints on when it can 
start improvement efforts. In this case, the college can start with the date on 
which it will begin to investigate the problem of practice and then map out 
the time line moving forward. 


Start with the time when it is feasible to implement changes and/or col- 
lect data. The rollout and testing of an improvement (the Do stage) may be 
the least flexible in terms of timing, as it may only make sense to pilot chang- 
es and/or collect data at certain points within the academic year. In this case, 
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Figure 3. Illustrative Time Line for Improvement Efforts in a College 


APR-AUG 
Identifying APR 
the Focus of 
l Scaling of 
mprovement 
: ; ; Improvements 
| | | | | | | | | | 
| : | | 
JULY-OCT OCT-JAN JAN-APR 
PDSA Cycle 1 PDSA Cycle 2 PDSA Cycle 3 
Plan Plan Plan 
< q q 
Act Do Act Do Act Do 
ay y 
Study Study Study 


NOTE: A worksheet for mapping out the time line for improvement efforts is provided in a supplement at the end of this 
toolkit. 


a college can determine when it will be able to carry out these activities in the Do 
stage and map forward and backward from there. 


Figure 3 contains an illustrative 12-month time line for improvement on a problem of prac- 
tice that builds in multiple PDSA cycles of 90 days or less. Understanding the problem and 
identifying improvements to be tested through PDSA cycles may take as little as a month 
or as much as six months depending on how experienced the team is with improvement 
and how much exploration is required to understand the problem of practice. It may be 
common for colleges to pilot changes and collect data based on the academic calendar, 
testing the improvements out during the fall or spring semesters (or both). Data collection 
should occur within the same semester to ensure rapid cycles. To allow for at least two 
iterative PDSA cycles within a year, colleges will need to be prepared to quickly assess the 
data and take action to define an objective for the next cycle (over the summer in 

Figure 3, for example). 
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Carrying Out Rapid-Cycle 
Evaluation 


When a college’s objective for improvement is focused on assessing effective- 
ness, whether it be the effectiveness of some aspect of a corequisite model, some 


instructional practice, or some new process for advising and scheduling, rapid-cy- 


cle evaluation can be a valuable tool. In this section, we start by describing the 
process for carrying out rapid-cycle evaluation. We then provide tips and tools 


for determining what to measure, determining which data to collect, and using 


comparison groups. 


The Rapid-Cycle Evaluation Process 


As a quick and focused version of traditional program evaluation, the process for 


carrying out rapid-cycle evaluation is relatively straightforward and likely to be 


familiar to colleges. We provide a brief description of the steps for carrying out a 
basic rapid-cycle evaluation below. 


Define the questions for the study. Just as was recommended for PDSA cycles, 


it is important to define a clear set of questions for the improvement cycle to 


A college will need to 
decide whether it wants 
to focus the study on 
outcomes or whether 
there are aspects of 
program delivery that 
might be important to 
measure to understand 
why outcomes did or did 
not change. 


ensure that the evaluation will generate the evidence needed for 
decisionmaking and that the questions can be answered within a 
short time frame. In rapid-cycle improvement, it can be useful to use 
a logic model to map out how the thing being evaluated is expected 
to lead to impacts, and this logic model can be used to generate the 
questions. We provide a more detailed description of logic models 
and an example of a logic model related to corequisite remediation 
in the next section (Tips and Tools: Determining What to Measure in 
a Rapid-Cycle Evaluation). A college will need to decide whether it 
wants to focus the study on outcomes or whether there are aspects 
of program delivery that might be important to measure to under- 
stand why outcomes did or did not change. For example, if a college 
were evaluating a new instructional approach for corequisite reme- 
diation students and found that it was ineffective or even harmful, it 
might be important for colleges to understand more about what was 
occurring in the classroom and how students perceived the instruc- 
tional approach in order to understand why it failed and avoid similar 
issues in the future. 
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Identify data and measures that will be collected to address the evaluation ques- 
tions. After identifying evaluation questions, the next step is to determine which data 
and measures will be used to address each of those questions. It is common for rap- 
id-cycle evaluations to rely on existing data sources and short-term outcome measures 
because of the need for data collection and analysis to be quick. However, there may 
be some potential to collect additional quantitative and/or qualitative data that will help 
colleges better assess the effectiveness and understand the results of particular coreq- 
uisite remediation features being tested. We describe different types of data that might 
be collected to inform rapid-cycle evaluation efforts (see Tips and Tools: Determining 
Which Data to Collect). 


Figure out how the comparison group will be chosen. Because the It is critical to ensure 
purpose of rapid-cycle evaluation is to measure impact, or effectiveness, 2 
it requires some sort of comparison group. For example, to test the effec- that those rolling out 
tiveness of a new corequisite remediation instructional approach, a col- the program have 

lege would want to roll out the instructional approach to students in one 50 ‘ 

set of classrooms and stick to “business as usual” in other classrooms sufficient guidance 
with similar students. Rapid-cycle evaluations commonly use randomiza- and resources. 

tion to assign students or classrooms to the treatment and comparison 

groups or use Statistical matching approaches to eliminate differences 

between the two groups of students. However, colleges may not have the capacity to 

carry out these approaches without support from external research partners, so they 

may choose comparison groups in less rigorous ways. We describe some options for 

comparison groups (see Tips and Tools: Using a Comparison Group). 


Deliver the program component being evaluated and collect data. The next step is 
to roll out the aspect of corequisite remediation that is being evaluated to the treatment 
group and collect data according to the predetermined plan. As described in the PDSA 
section, itis critical to ensure that those rolling out the program have sufficient guid- 
ance and resources and are rolling out that aspect of corequisite remediation in its ideal 
form to ensure that the evaluation is examining effectiveness when “done right.” In the 
case of randomized approaches, it is also important to prevent those in the comparison 
group from receiving the corequisite remediation feature being tested. To ensure rapid 
cycles of improvement, it is important that these activities be able to take place within a 
single semester. 


Conduct analysis. In rapid-cycle evaluation, the focus of analysis is on comparing 
outcomes for the group receiving the corequisite model component, process, or prac- 
tice being tested with a comparison group. The first step of analysis is understanding 
whether there are differences between the two groups. If a college uses a lottery to 
assign classrooms or students to the two groups, students and instructors should look 
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Rapid-cycle 


similar across easy-to-access characteristics (e.g., race and ethnicity, incoming 
test scores, teaching experience). That may not be the case if students or class- 
rooms are chosen in a different way (e.g., instructor or student volunteers). For 
colleges with sufficient capacity or a willingness to bring on external research 
partners, statistical matching techniques can help address some of the differenc- 
es across the comparison group and the group testing out the corequisite model 
component. We provide some additional detail on this topic (see Tips and Tools: 
Using a Comparison Group). 


Make decisions based on the findings. Once a college has evidence on the 

effectiveness of the corequisite model component, process, or practice, the next 
step is to make a decision about how to move forward. Colleges may 
choose to scale if the evaluation showed evidence of effectiveness, or 
they may choose to modify or eliminate the corequisite model component 


evaluation is distinct being tested if the results were less favorable. Whatever the decision, 


in its focus on 
comparison groups 
and more-rigorous 
data analysis. 


colleges should consider best practices for building buy-in and ensuring 
action on the findings, including some of the valuable lessons offered 
from the field of quality improvement on building consensus around the 
objective of the evaluation and reflecting on findings with a broad group 
of relevant college staff. 


Continue cycles of improvement. A college must then consider what 

might be addressed in future cycles of improvement. Some rapid-cycle 

evaluation efforts engage in multiple cycles of evaluation for the same 
intervention as it is gradually scaled to larger and larger groups of students. 
Alternatively, the college might want to shift to quality improvement as it scales 
the new practice to allow for continued refinement of how it is delivered. Once 
the improvement team decides that the problem has been sufficiently addressed 
through the success of improvement efforts and evidence of effectiveness, the 
college can shift focus of improvement to another problem of practice related to 
corequisite remediation. 


The rapid-cycle evaluation process may sound familiar because there are many 
similarities in how rapid-cycle evaluation and PDSA cycles carry out testing 

and evaluation (Figure 4). However, the field of quality improvement provides 
substantially more guidance on processes for planning improvements, building 
teams, and reflecting on findings. Rapid-cycle evaluation is distinct in its focus on 
comparison groups and more-rigorous data analysis. 
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Figure 4. Overlap in the Steps to Carrying Out Rapid-Cycle Evaluation and PDSA Cycles 


Rapid-Cycle Evaluation Quality Improvement 


Define evaluation questions 


Identify questions and make 
predictions 


+ Determine data and measures ——> Plan 
+ Identify comparison group * Define activities 
- Roles and responsibilities 
« Time line 
¢ Roll out program component ¢ Deliver program with an 
to allow evaluation —? improvement that is being Do 
- Collect data tested 
* Collect data 
- Analyze data —— _.- Analyze data 


Share findings and reflect Study 


Document findings 


Make decisions 


Identify possible actions 


Continue cycles 


Plan for improvement Act 


Share findings broadly 


Continue cycles 


Tips and Tools: Determining What to Measure ina 
Rapid-Cycle Evaluation 


Given the short time lines involved with rapid-cycle evaluation, it may be important 

to focus only on the evaluation questions and measures that are most important to 
understanding effectiveness. To determine what to measure, evaluators often use a 
logic model. A logic model maps out how a particular practice or aspect of a corequisite 
model is believed to be related to student success outcomes. Logic models typically 
have at least three sections, including the following: 


Inputs are the resources and supports required for delivering the aspect of coreq- 
uisite remediation that is being evaluated. These could include staff, instructional 
materials, financial resources, and facilities. 


Program activities are the core components of the corequisite model, i.e., the 
things that are being done to deliver the particular practice or aspect of the coreq- 
uisite model being evaluated. Depending on which part of corequisite remediation 
is being evaluated through rapid-cycle evaluation, these activities might include 
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structural components of the corequisite model, instructional activities, or 
other required support activities (e.g., scheduling, advising, tutoring). Some 
logic models also specify outputs, which are the immediate results of hav- 
ing delivered the corequisite remediation practice or model component as 
planned. Example outputs include the number of students enrolled properly, 
the hours of instruction received, instructional strategies used, and portfolios 
of classroom work. 


Outcomes represent the types of expected changes and results when the 
corequisite is delivered successfully. It can be useful to break these out into 
short-term outcomes, which might be observed within the semester, and lon- 
ger-term outcomes, which ultimately are the measures of student success that 
the feature, practice, or process is meant to affect. Because of the need for 
quickly turning around results, rapid-cycle evaluation will focus on measuring 
short-term outcomes. 


If the fictional college in Box B wanted to carry out a rapid-cycle evaluation to 
assess the effectiveness of its new two-instructor alignment approach, it might 
develop a logic model like the one presented in Figure 5. Although many other 


Figure 5. A Logic Model for a College Aligning Instruction Across Two-Instructor Pairs 


Inputs Program Short-Term Intermediate Long-Term 
. Instructor pairs Activities Outcomes Outcomes Outcomes 
» Alignment plan ¢ Instructor pairs ¢ Daily instruc- ¢ Students learn + Students persist 
oe meet for tion is more more 
+ Training and annie ann « Students earn 
uidanee u 9 - Students feel more credits 
; >a Instructors align >a Support pro- ? better supported 
eine (or shared syllabi vided is more eae 
planning Mpa + Students pass complete 
calendars useful courses 
« Guidelines for SECS 
g + Instructors « Students make 
syllabi, , 
deliver stronger con- 
schedules aye : 
corequisites nections 
« Other 
corequisite 
materials 


T t t T 


Facilitators and Barriers 


¢ Structure of departments within the college (i.e., siloing), leadership support, funding, instructor attitudes 
about pairing and collaboration, training and guidance, time, etc. 


NOTE: A worksheet for developing a logic model is provided in a supplement at the end of this toolkit. 
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activities are required to deliver the corequisite model, the college is only evaluating 
the modification requiring two-instructor meetings and syllabi alignment, so Figure 5 
only covers the specific aspects of corequisite remediation that are being evaluated. 
The items in bold illustrate what the college might want to measure in its rapid-cycle 
evaluation given limited time and resources. 


Tips and Tools: Determining Which Data to Collect 


There are several decisions that colleges must make when planning for data collection. 
This section outlines these data collection decisions and provides examples of the 
different types of data that colleges might consider collecting to inform improvement 
efforts. This guidance on data collection may also be useful for informing data collection 
approaches under quality improvement and PDSA cycles. 


As described above, the types of data collected and decisions about how to analyze 
those data will vary according to the specific questions that a college is 


trying to answer. A college’s resources and capacity to collect data and The types of data 
the short time frame for improvement cycles may also place limitations on 

what can be collected; rapid-cycle evaluations often rely on existing data. collected and 

In addition to these considerations, colleges will need to determine the decisions about 


following when designing plans for data collection and analysis: 


how to analyze 
those data will vary 


¢ Types of data to use: There are several types of data that can be 
used to inform improvement. Table 4 provides a list of data sources 


and some information to help colleges choose the appropriate data according to the 
sources for improvement efforts. Rapid-cycle evaluation requires some = : 
measure of effectiveness; therefore, the college may need to rely specific questions 


on administrative data, existing surveys, or in-class assessments to ° 
examine outcomes. In some cases, colleges may also want to have a that a college IS 
deeper understanding of why particular outcomes were observed, so trying answer. 
they may want to collect data on perspectives from key stakeholders or 

on barriers and facilitators to achieving outcomes. In these cases, the 

college may benefit from collecting qualitative data, such as through 

interviews or focus groups. 


Data from a sample versus the full population: Data are often collected from just 
a sample of individuals (rather than the full population) in order to cut down on the 
costs of data collection and the burden on participants and to improve the quality 

of the data that is collected. However, collecting data from a sample also has limita- 
tions. If a college is using quantitative data and testing for statistical significance, the 
sample must be large enough to do so. In addition, how that sample is chosen may 
limit the degree to which the findings can represent the overall population. If it is 
important that the data from the sample represent the population, it would be optimal 
to choose the sample of individuals randomly, if possible. 
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Create new measures or use ones developed by others: Wherever possi- 
ble, it can be useful to use survey questions and scales that have been de- 
veloped by other colleges or researchers. Using existing measures can save 
a college time in mapping out a data collection plan, can make the evidence 
more relevant to other colleges, and can help ensure that the measure has 
been proven to be of high quality with other data. However, there may not be 
other studies that have looked at the concepts a college is interested in; in 
these cases, the college will need to develop new measures. 


Whether existing data sources can be used: Because data are costly and 
take time to collect, it can be beneficial to rely on existing data sources wher- 
ever possible, and rapid-cycle evaluations often rely entirely on existing data. 
If students or instructors are already surveyed or brought together for other 
reasons, it may be easier to add a few questions to these existing surveys 

or to pull together instructors for focus groups during department meetings, 
rather than creating separate data collection efforts. If improvement cycles 
are taking place across the college, those teams should work closely to coor- 
dinate their efforts to ensure that data collection is not overburdening specific 


groups of students or college staff. 


Table 4. Example Data Sources and Their Merits and Limitations 


Data Source 
Administrative 
data 


Merits 


¢ Providing systematic data across broad 
groups of students, college staff 


Measuring academic success outcomes 
for students 


Accounting for differences in student, 
classroom, and instructor characteristics 
in analysis 


Limiting burden on students or school 
staff participants 


Limitations 

« Limited to the measures already being 
collected for other purposes; can be hard 
to add new information 


May take time to process and be ready for 
use 


Surveys 


Providing systematic data across broad 
groups of students, college staff 


Providing data on how a program is being 
rolled out, perspectives of individuals, 
successes and challenges, and explana- 
tions for trends in quantitative data 


Assessing outcomes not captured in 
administrative data 


Can take a moderate amount of time and 
resources if creating new surveys 


« May require institutional review board 
(IRB) approval 


Can suffer from low response rates that 
limit representativeness 


May require incentives to secure partici- 
pants 


Accuracy of self-reported data can be 
affected by response bias and respondent 
engagement 
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Data Source 
Focus groups 


Merits 


Providing in-depth data from a moderate 
number of individuals 


Providing data on how a program is being 
rolled out, perspectives of individuals, 
successes and challenges, and explana- 
tions for trends in quantitative data 


Providing a setting where respondents 
can interact and build off of each other’s 
ideas 


Limitations 


Can take a moderate amount of time and 
resources 

May require IRB approval 

Cover a moderate number of individuals 
and may not be representative 

May require incentives to secure partici- 
pants 

Accuracy of self-reported data can be 
affected by response bias and respondent 
engagement and interactions 


Interviews 


Providing in-depth data on individual 
perceptions and knowledge 


Providing data on how a program is being 
rolled out, perspectives of individuals, 
successes and challenges, and explana- 
tions for trends in quantitative data 
Providing individuals with privacy to dis- 
cuss sensitive issues and perspectives 


Can take substantial time and resources 
May require IRB approval 


Cover few individuals and may not be 
representative 


May require incentives to secure partici- 
pants 
Accuracy of self-reported data can be 


affected by response bias and respondent 
engagement 


Observations 


Providing data on how a program is being 
rolled out 

Assessing interactions between various 
groups 


Can take substantial time and resources 


May cover few individuals and may not be 
representative 


Can be challenging to collect reliable and 
consistent data across multiple observers 


May require IRB approval 


Program docu- 
ments 


Measuring the requirements for a course 
or program 

Understanding how a program is intend- 
ed to be implemented 

Collecting data with limited burden on 
students or school staff participants 


Information on measures of interest may 
be limited and not systematic across sec- 
tions, programs 


Program data 
(e.g., sign-in 
sheets, time 
logs, advisor 
and instructor 
notes) 


Providing data on how a program is being 
rolled out 


Collecting data with limited burden on 
students or school staff participants (if 
data already have been collected for 
other purposes) 


Can be time-consuming to collect (if not 
already collected for other reasons) 
Accuracy of self-reported data can be 
affected by response bias and respondent 
engagement 


NOTE: A worksheet for identifying data sources that address questions for a cycle is provided in a supplement at the end of 


this toolkit. 
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Another important consideration for colleges is how to ensure that the data 
collected are of high quality. If the data collected do not accurately reflect what is 
happening on the ground or the perceptions of those involved, the actions taken 
based on those data may not actually lead to improvements. There are many 
strategies for ensuring that the data collected are high quality, but we offer the 
following four tips: 


Pilot data collection tools. If there is sufficient time in the planning stage, it 

can be useful to pilot a data collection tool, such as a new survey, an assess- 

ment, or questions for a focus group. This pilot will help colleges identify 

effective and ineffective portions of the data collection tool and streamline 
and make refinements. 


¢ Build a safe environment for data collection. It is important to ensure 


Another important 
consideration for 
colleges is how to 
ensure that the data 
collected are of high 
quality. 


that respondents feel comfortable reporting data accurately. The college 
can reassure participants that data collected for improvement efforts 
will be used only for the purposes of improvement and never to evaluate 
staff or hold people accountable. Collecting data anonymously is anoth- 
er way to increase the comfort of participants. 


¢ Communicate the importance of data collection. Colleges should 
inform the subjects of data collection about the value of the data they 
are providing and how those data will be used to help the subjects. This 
can help build buy-in, and subjects will be more likely to devote time 


and effort to providing accurate data if they see the data collection as 
valuable. 


Include data and research experts on the team. Figuring out how to collect 
high-quality data can be difficult, and improvement teams can benefit from 
members with expertise on data collection and research, including members 
of the institutional research office, other data experts from within the college, 
or external consultants. 


In addition to considering strategies for ensuring high-quality data, colleges 
should make sure to comply with human subject protection policies and Family 
Educational Rights and Privacy Act requirements that protect student data. Ifa 
college does not have its own institutional review board, it may need to work with 
an external review board to ensure approval for the data collection procedures. 
Colleges should consider ways to ensure regular and expedited review processes 
for ongoing improvement efforts given the need for short cycles of improvement, 
and improvement teams should adjust the time line to account for these review 
processes. 


Tips and Tools: Using a Comparison Group 


The ultimate goal for rapid-cycle evaluation is to measure effectiveness—i.e., 
the impact caused by a particular practice or corequisite model component. To 
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measure the impact of something, it is essential to have a comparison group in order 

to compare outcomes for students and instructors who receive the practice or program 
component with similar students and instructors who do not. For example, if a college 
wanted to assess the importance of class size in corequisite courses, it would need to 
compare corequisite sections of smaller class sizes with a comparison group of larger 
class sizes (or vice versa). Without a comparison group, it is hard to know what is driving 
the patterns observed in the data, as it could be other programs or initiatives in the 
college or changes in the student population over time. 


To illustrate the importance of a comparison group for measuring It is essential to have a 
impacts, we can use the example provided in Box B. Because this 0 0 

college was engaging in quality improvement rather than rapid- com pa rison g roup in order 
cycle evaluation, there were no efforts to create a comparison to com pare outcomes for 
gen: sie assume that pa rates for the algebra instructors students a nd instructors 
testing the improvement increased from 51 percent to 72 percent, 

and the improvement team concluded that the four hours of time who receive the practice 
that instructor pairs spent together was an improvement. But or prog ram com ponent 
suppose the college also rolled out a new early alert system that : Fr 

fall, making it unclear how much of the improvement in outcomes with similar students and 
is due to the early alert system rather than the improvement instructors who do not. 
strategy. In addition, there may have been differences between 

last year’s students and this year’s students that also contributed 

to success. If the improvement team members really wanted to measure whether the 

improvement being tested caused the 11-percentage-point increase in passing rates 

(rather than the early alert system or differences in the student population), they would 

have needed to develop a comparison group. 


There are several possible approaches to choosing a comparison group. We describe 
the three main options below and outline some of the limitations of using less rigorous 
ways to choose comparison groups in Table 5. 


Randomization (i.e., a lottery). The ideal strategy for choosing a comparison group for 
rapid-cycle evaluation, where possible, is to choose students and staff through a lottery 
(also referred to as randomization). This is the best way to guarantee that the compar- 
ison group is going to be similar to the group testing out the new program or improve- 
ment strategy, including characteristics that are observable in data (e.g., age), and char- 
acteristics that are not (e.g., motivation). Lotteries can also be valuable because they 
offer a fair way to figure out who gets the opportunity to try something first. However, 

it can often be difficult or infeasible to assign students or instructors to a comparison 
group through a lottery. In addition, lotteries can be counterproductive to improvement 
efforts if faculty or other college staff are unhappy about the process. In such cases, 
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colleges will have to consider other options for creating a comparison group. 


Statistical approaches. Using statistical methods, such as matching or regres- 
sion analysis, can help account for the observable differences in students (e.g., 
Pell-eligible, test scores) and instructors (e.g., adjunct or full time, teaching 
experience). These approaches can help eliminate some of the concerns about 
differences between the students and/or instructors in the pilot group and the 
comparison group driving the results. However, there may still be unobservable 
differences that cannot be accounted for if the college does not have data on 
them (e.g., student motivation, instructor quality). The use of matching or regres- 
sion methods may require colleges to bring on external partners with expertise in 
data analysis. 


Other options for comparison groups. Although rapid-cycle evaluation typically 
calls for more-rigorous methods, such as using a lottery or statistical methods, 
there are other types of comparison groups that colleges often rely on when 
piloting new programs and assessing data (Table 5). However, there are important 
limitations that colleges need to be aware of with each of these comparison group 
options. Because of these limitations, colleges should be careful not to assume 
that the findings of the rapid-cycle evaluation provide definitive evidence of effec- 
tiveness unless randomization or statistical approaches are used. 


Table 5. Options for Comparison Groups and Limitations when Rigorous Methods Are Not Possible 


Comparison Group Option Limitations 

Students, instructors within the same « Those who were selected or volunteered might be different 
campus or college who did not volunteer from those piloting in important ways, and those differences 
to deliver the program or pilot may drive differences in outcomes. 

Students, instructors from other campus- | « Differences in other programs and initiatives across campuses 
es not delivering the program or piloting and/or differences in the student and instructor population 
the improvement could be driving results. 


Students, instructors from other colleges May be challenging to access data from other colleges 


not delivering the program or piloting the Differences in other programs and initiatives across colleges 
improvement and/or differences in the student and instructor population 
could be driving results. 


Prior cohorts of students before the Changes to other programs and initiatives over time and/or 


program was delivered or improvement differences in the student population could be driving results. 

was piloted 

Pretest and posttest for students, instruc- | « Change in scores may have happened even without the new 

tors receiving the practice or program practice or program component 

component « Assessments may not be great predictors of long-term student 
SUCCESS. 
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Conclusion: Ensuring Success in 
Improvement Efforts 


Throughout this toolkit, we have repeatedly discussed some of the conditions that help 
make improvement efforts work well and ensure that various activities lead to real 
improvements on the ground. To conclude, we emphasize three of the most important 
conditions that help drive action toward improvement. 


A committed team with time to engage.”° Improvement efforts (especially in the case 
of quality improvement) rely on the ongoing and frequent involvement of practitioners. 
We noted that it is critical for an improvement team to be able to meet frequently to 
discuss progress and reflect on the results of the process. Data collection and the 
design of potential improvements also take substantial time. Although we recom- 
mended that this core team be kept relatively small to ensure it is agile and sufficiently 
resourced, the team should find ways to incorporate the voices of a broader set of 
individuals in the college throughout the process. Broad groups of stakeholders should 
be included in the process of exploring the problem, identifying drivers and possible 
improvements, reflecting on findings, and determining how to move forward. A more 
democratic approach to decisionmaking can help guide the focus of improvement to 
the most-important issues, provide critical insights on how to address those issues, and 
ensure the buy-in needed to drive collective action. 


Support and involvement of leadership.”' Leadership involvement and support is im- 
portant because it communicates to all of those involved that improvement is a priority. 
In addition, leadership can help ensure that the improvement team is equipped with the 
resources needed to carry out the improvement process, as well as the resources and 
authority to scale improvements in response to findings from the process. Leadership 
can also play an important role in setting the open and honest tone for improvement 
efforts and negotiating the different priorities and perspectives of those involved in the 
improvement process. 


An openness to inquiry and reflection.?2 College improvement efforts are built on the 
belief that educational programs and initiatives need to be improved and that doing 

so requires taking a critical, evidence-based look at these programs and initiatives. 
Involved individuals must therefore be willing to challenge their beliefs about what 
works and what does not, willing to collect additional data on what is happening on the 
ground, and open to reflecting on data that might not always be positive. Leadership 
and the improvement team can help create an environment that supports this openness 
to inquiry in various ways, such as by ensuring that the evidence collected for improve- 
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ment is not used for staff evaluations or other high-stakes purposes, requiring that 
individuals be equally valued and respected throughout the process, and letting 
staff play a key role in setting the agenda for improvement cycles. The short 
nature of improvement cycles and the testing of new improvements with small 
groups of students or instructors can limit the risk of trying new things, providing 
a space that offers the “freedom to fail.” 
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Quality Improvement and Rapid-Cycle Evaluation: 


Prioritizing Problems of Practice 


In the toolkit, we described how improvement efforts typically focus on a single problem of practice. This work- 


sheet aims to help colleges prioritize among a group of problems that a college might be considering for improve- 


ment efforts. 


Step 1. List the major challenges or issues that the college wants to resolve. Based on the input of a broad range 


of stakeholders within the college and using existing evidence, identify issues or areas of uncertainty that the 
college is facing with corequisite remediation and list these problems in the first column. 


Step 2. Rate problems according to the four criteria provided. Choose a rating from 1 (low) to 5 (high) and provide 


a brief description of the rating. These ratings can be determined as a group through discussion or done individu- 
ally and aggregated across team members. 


Step 3. Choose a problem of practice for improvement efforts to focus on. Based on individual or aggregated 


group ratings, determine which of the problems of practice should be prioritized for improvement efforts. 


student success 
in corequisite 
models taught by 
instructor pairs 


Evidence: 

Lower student 
outcomes in 
two-instructor 
sections; faculty 
perceptions of 
alignment issues 


Reasoning: 

Much that can be 
done to improve 
issues, but some 
drivers of issues 
out of the col- 
lege’s control 


Reasoning: 
Planning to scale 
two-instructor 
model soon, so 
needs to be re- 
solved soon and 
will affect many 
students 


Reasoning: 
Faculty and ad- 
ministrators view 
as very import- 
ant; support staff 
and instructors 
in one-instructor 
models less so 


Problem of Practice | Actionable Aligned with Mission | Consensus on Im- Evidence Limitations 

(PoP) and Support- portance 

ing Evidence PoP is entirely driven | Addressing PoP will Substantial existing 
by things under the cause big changes All stakeholders evidence on how to 
college’s control (5) to | in student success agree itis important | address the PoP (5) to 
entirely outside of the | (5) to no changes in to address the PoP no evidence available 
college’s control (1) student success (1) (5) to few agree itis | (1) 

important (1) 
Problem: Rating (1-5): 3 Rating (1-5): 5 Rating (1-5): 4 Rating (1-5): 4 
Issues with 


Reasoning: 

Many colleges 
struggling with 
this, but only 
anecdotal evi- 
dence on how to 
address it 
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Quality Improvement and Rapid-Cycle Evaluation: 


Prioritizing Problems of Practice 


Problem: Rating (1-5): Rating (1-5): Rating (1-5): Rating (1-5): 
Reasoning: Reasoning: Reasoning: Reasoning: 

Evidence: 

Problem: Rating (1-5): Rating (1-5): Rating (1-5): Rating (1-5): 
Reasoning: Reasoning: Reasoning: Reasoning: 

Evidence: 

Problem: Rating (1-5): Rating (1-5): Rating (1-5): Rating (1-5): 
Reasoning: Reasoning: Reasoning: Reasoning: 

Evidence: 
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Quality Improvement: Using a Fishbone Diagram to Understand the 
Problem and Identify Objectives for Improvement 


In the toolkit, we described the need to deeply understand the problem of practice for the purposes of quality 
improvement efforts through root cause analysis. Root cause analysis can help the college identify where im- 
provement is needed and develop more-successful objectives for improvement that can be tested through a Plan- 
Do-Study-Act (PDSA) cycle. This worksheet aims to help colleges identify root causes through a cause-and-effect 
diagram, or a fishbone diagram 


Step 1. Write the problem of practice related to corequisite remediation at the mouth of the fish. 


Step 2. Figure out what categories of things might contribute to a particular problem. One possible set of catego- 
ries could include students, college staff, rules, policies and procedures, and facilities and supplies. 


Step 3. Ask “Why does this happen?” to identify different causes of the problem, and record these causes on the 
“bones” of the fish. 


Step 4. Identify subcauses underlying the initial set of causes recorded in the diagram, and keep breaking the 
issue down until participants cannot think of any additional causes. 


Step 5. Select (or rank) the branches where stakeholders see the most potential for improvement (i.e., changes 
can be made in how things are done, and these changes are likely to help address the particular issue), and list 
out specific improvements that can be tested through improvement cycles. 


Consider using sticky notes and other engagement strategies to solicit broad input and systematic ratings if the 
group conducting root cause analysis is larger. Note, an example of a completed fishbone diagram is provided 
in Figure 1 of the toolkit. 
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Quality Improvement: Using a Fishbone Diagram to Understand the Problem and Identify Objectives for Improvement 


PROBLEM 


Category 2 
Category 4 


Category 1 
Category 3 
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Quality Improvement: Defining Questions, Measures, 
and Predictions for a Cycle 


In the toolkit, we described how colleges engaging in quality improvement will need to identify specific questions 
and make predictions about what will happen when a specific improvement is tested through the PDSA cycle. 

In addition, it is important to identify what measures will be used to test those predictions and which data are 
required to track that measure. Quality improvement typically calls for colleges to focus on a set of measures 

that will be used across multiple PDSA cycles to ensure that comparisons can be made across different types of 
improvements being tested. This worksheet will help colleges identify questions, determine the data sources and 
measures that will be used to address questions, and make specific predictions about what will be found. 


Step 1. Develop concrete questions that will be addressed through the evaluation. What are the key things 
that the college needs to know to determine whether the change being tested was an improvement? These items 
should be turned into specific, measurable questions. 


Step 2. Identify data sources and measures. The college should consider which types of measures it needs to 
answer the proposed questions. Some of these measures should be drawn from a common set of measures that 
can be used across PDSA cycles, while others might be specific to a particular improvement being tested. It is also 
important to think about data and measures that can be collected and assessed by practitioners within a relatively 
short period. 


Step 3. Make a prediction. Next, make a specific prediction about what the improvement team expects the 
measure to show during that PDSA cycle. If the data is being collected only from those testing the improvement, 
this prediction should be a benchmark, a specific value that the measure is expected to reach, or the amount of 
progress made on the measure. If the college has access to similar individuals not testing the improvement and 
will also have data on the measure for this group, the prediction can focus on a comparison of the two groups. 


Question Data Source Measure Prediction 

How many hours will instructor Weekly instructor Hours of time collaborating Instructors will meet for at least 

pairs subject to the new alignment | survey by phone, email four hours. 

strategy meet for in order to align 

syllabi? 

Will students be more likely to End-of-course satisfac- | Likert question: “The devel- Students in courses with the in- 

perceive that the developmental tion survey opmental education support | structor pairs testing the alignment 

education part of the corequisite was important to supporting | strategy will be at least 10 percent- 

model is valuable? my success in the college age points more likely to agree 

course.” or strongly agree compared with 

other students. 

Will student success rates in the Administrative data Percentage of students Course passing rates will increase 

college course increase? passing by 5 percentage points relative to 
prior semesters. 
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Quality Improvement: Defining Questions, Measures, and Predictions for a Cycle 


Question Data Source Measure Prediction 
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Rapid-Cycle Evaluation: Using a Logic Model to Guide the Evaluation 
Questions and Measures 


In the toolkit, we recommended that colleges identify specific questions to guide evidence-building activities 
and suggested that a logic model might be a useful tool for developing these questions. This worksheet will help 
colleges develop a logic model and identify evidence-building questions for a continuous improvement cycle. 


Step 1. Fill in the sections of the logic model. Start with the key activities, including what must be done to 
deliver the corequisite model component, practice, or process that is evaluated. Next, list the inputs—i.e., what re- 
sources are required to achieve these activities. Then fill in long-term outcomes—i.e., the ultimate student success 
outcomes the college aims for. Map backward to shorter-term outcomes to describe what the college hopes will 
happen between the time the corequisite activities take place and the long-term outcomes are observed. 


Step 2. Identify which aspects of the logic model are highest priority to measure effectiveness and inform 
improvements. Because the goal of rapid-cycle evaluation is to assess effectiveness, a college will definitely 
want to measure outcomes. Short-term outcomes may be the only ones observable within the short time frame for 
the cycle. If the college decides it is important to measure how things worked and to understand more about the 
role of the college’s context, it may want to also examine whether and how the key activities were carried out and 
facilitators and barriers to success. 


Step 3. Develop concrete questions that will be addressed through the evaluation. Once you have identified 
the things you want to measure to assess effectiveness (and other aspects of program delivery), these items need 
to be turned into questions. For example, were students who received a corequisite remediation practice more 
likely to pass the college course? How often did instructors use the practice being evaluated? What are the barri- 
ers instructors faced to using the practice? 


Note, an example of a completed logic model diagram is provided in Figure 5 of the toolkit. 


Inputs 


What resources are need- 
ed to accomplish the key 
activities? 


Program Activities 


What are the things that 
must be done to deliver the 
corequisite as planned? 


Short-Term Outcomes 


What should happen imme- 
diately if the corequisite is 
delivered as planned? 


Intermediate Outcomes 


What are the outcomes 
expected within 1-2 se- 
mesters if the corequisite 
is successful? 


I 


I 


Long-Term Outcomes 


What are the ultimate 
outcomes that the college 
is aiming for? 


Facilitators and Barriers 


What are the things that can help or hinder the success of achieving the key activities and intended outcomes? 


Research questions: 
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Rapid-Cycle Evaluation: Developing a Data Collection and Analysis 
Plan 


In the toolkit, we recommended that colleges choose the best sources of data to address particular evaluation 
questions and to lay out a plan for how data collection will take place. This worksheet will help colleges develop 
that plan. 


Step 1: Determine which data should be collected. Identify the best sources of data for each evaluation ques- 
tion, whether this is quantitative data from administrative records or surveys; qualitative data from focus groups, 
interviews, observations, or surveys; or program data from program documentation and data records. Determine 
how many individuals data will be collected from (i.e., all individuals participating or a sample), and whether ex- 
isting measures will be used or new ones will be developed. Note: Table 4 in the toolkit provides some info on 
pros and cons of different data sources and some tips for ensuring high-quality data. 


Step 2: Determine how to construct the comparison group. Because rapid-cycle evaluation aims to produce ev- 
idence of effectiveness, it is important to have a comparison group to be able to measure this effect when looking 
at outcomes. For other types of measures on how the program was rolled out and whether there were barriers and 
facilitators, it may be less important to have comparison group data. For each measure, describe whether there 
will be a comparison group and how the comparison group will be constructed. Note: As you consider different 
comparison groups, Table 5 in the toolkit provides some important limitations to keep in mind. 


Strategies for ensuring rapid evaluation cycles: 
1. Will you primarily be relying on existing data sources? 
2. Will you be measuring things that can be observed within a short period? 


Strategies for ensuring high-quality data: 

1. Will you pilot new questions or instruments to ensure that they will produce the data you intend for them to 
produce? 

2. Will you collect data anonymously and inform participants about who will have access to responses to in- 
crease the likelihood they will provide honest and accurate responses? 

3. Will you provide participants with information on how the data will be used to communicate the value of 
investing time in data collection and providing honest and accurate responses? 
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MODELS 


Rapid-Cycle Evaluation: Developing a Data Collection and Analysis Plan 


Evidence-building Data source(s) Who and how many Measures and instru- Comparison group 
question in the data collection ments used 

sample 
Did alignment strategy Student survey All students enrolled in Add question(s) to exist- | Sections that have in- 


(requirement to meet 
four hours and align 
syllabi) lead to greater 
alignment of instruction? 


two-instructor sections 
of the corequisite 


ing student satisfaction 


survey 


structor pairs who were 
not selected by lottery 
to pilot the alignment 
strategy 
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Quality Improvement and Rapid-Cycle Evaluation: Assigning Roles and 
Responsibilities 


In the toolkit, we recommended that colleges assign clear roles and responsibilities for each activity in the Plan, 
Do, Study, and Act stages of the continuous improvement cycle. This worksheet will help colleges identify task 
leads and identify roles and responsibilities for all individuals involved in a continuous improvement activity 


Step 1. Identify the detailed set of activities for each stage of the cycle. Start by listing the detailed activ- 
ities for each stage of the cycle that will help achieve the objectives identified for the cycle and support the 
evidence-building efforts required. Note: Figure 1 in the toolkit lays out some general categories of activities 
for each stage that can help you think through the types of activities that should be listed. 


Step 2. Assign a core improvement team member to lead each activity. This activity lead will be the person to 
provide guidance to other participants in the activity, make sure the activity stays on track and on time, and report 
back on progress to the improvement team. If you do not have enough people or the right people on the core 
team to oversee activities, consider adding others to the team. 


Step 3. Identify others who will be involved in the activity and define their roles and responsibilities in par- 
ticipation. Other participants may include individuals helping the activity lead plan for and oversee the activity, 
participants in data collection, individuals who will be consulted on findings and/or plans for action, and individu- 


als who must be involved in scaling improvements. 


Activity Activity Lead Administrators | Faculty Data and Analysis Others (e.g., ad- 
Experts visors, support 
staff, students) 
Stage: PLAN Who: Faculty chair Who: N/A Who: Six fac- | Who: Information Who: N/A 
; ulty repre- _| retrieval represen- 
Activity: Refine What: Create plan to | What: peniaiwes | <attve What: 
alignment strat- | get feedback, iden- 
egy through tify faculty to give What: Give What: Consult on 
discussions with | feedback, refine feedback, plan to get feed- 
faculty reps Strategy, report back approve back 
refinements 
to strategy 


Activity Activity Lead Administrators Faculty Data and Analysis Others (e.g., advisors, 
Experts support staff, students) 
Stage: Who: Who: Who: Who: Who: 
Activity: What: What: What: What: What: 
Stage: Who: Who: Who: Who: Who: 
Activity: What: What: What: What: What: 
Stage: Who: Who: Who: Who: Who: 
Activity: What: What: What: What: What: 
Stage: Who: Who: Who: Who: Who: 
Activity: What: What: What: What: What: 
Stage: Who: Who: Who: Who: Who: 
Activity: What: What: What: What: What: 
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Quality Improvement and Rapid-Cycle Evaluation: Determining Time 
Lines for the Improvement Cycle 


In the toolkit, we recommended that colleges map out clear time lines for activities to ensure that things stay on 
track through rapid cycles of improvement. 


Step 1. Identify the detailed set of activities for each stage of the cycle. Start by listing the detailed activi- 
ties for each stage of the cycle that will help to achieve the objectives identified for the cycle and support the 
evidence-building efforts required. Note: Figure 1 in the toolkit lays out some general categories of activities 
for each stage that can help you think through the types of activities that should be listed. 


Step 2. Assign a start and end date for each activity. In the toolkit, we recommended three potential approach- 
es for mapping out time lines: (1) starting with when a college would like to see action and mapping backward, 

(2) starting with when a college can begin planning and mapping forward, or (3) starting with when a program will 
be delivered and data collected and mapping forward and backward. 


Step 3. Develop one or more key deadlines at which the lead will report on progress. Assigning deadlines, or 
key check-in points at which the activity lead will assess progress and report back to the group, will help ensure 
that activities stay on track and will provide an opportunity for adjusting time lines if necessary. 


Activity Start Date End Date Deadline 1 Deadline 2 Deadline 3 
Stage: PLAN June1,2019 | July15, 2019 | Date: June 10, | Date: June 30, | Date: July 
; 2019 2019 15, 2019 

Activity: Refine 

alignment strategy What: Finalize | What: Finish What: Final- 

through discussions plan to get | collecting ize plan 

with faculty repre- feedback feedback for refined 

sentatives from faculty | alignment 
representa- | strategy 
tives 


Activity Start Date End Date Deadline 1 Deadline 2 Deadline 3 
Stage: Date: Date: Date: 
Activity: What: What: What: 
Stage: Date: Date: Date: 
Activity: What: What: What: 
Stage: Date: Date: Date: 
Activity: What: What: What: 
Stage: Date: Date: Date: 
Activity: What: What: What: 
Stage: Date: Date: Date: 
Activity: What: What: What: 
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TL-319-IES 


n education, practitioners are increasingly looking for strategies to 
bring data and evidence into decisionmaking and to roll out programs 
in intentional ways that allow for ongoing improvement. This toolkit 
acts as a guide to improvement strategies for teams of practitioners 
at colleges, providing them with the knowledge and tools they need to 
carry out rapid cycles of data-driven improvement. It provides an overview 
of key strategies for making quick improvements to education programs 


and some tools and tips for carrying out these efforts on the ground. An 


included supplement provides seven worksheets that colleges can use to 
plan for a rapid cycle of improvement. 


The authors focus on corequisite remediation, a common approach 
being adopted by colleges. Corequisite remediation shifts the way that 
developmental education is provided so that students directly enter a 
college-level course and receive additional academic support during 
that same semester to address the challenges they face with basic math, 
reading, or writing concepts. 


This toolkit was developed for improvement teams made up of 
administrators, department and faculty chairs, and other college staff 
who are helping to oversee important initiatives and programs, such 

as corequisite remediation, and who are looking to build an ongoing 
process for addressing challenges they face and improving the way that 
corequisite models are provided through data use and reflection. 


EDUCATION AND LABOR 


